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ABSTRACT ^ «go-go years", the late 50«s 

and 6C»s when f-deral and foundation funding of education increased 
so much? 3as a hage increase in the number of i^^^^^^Ji^^ti if ^ 

(both print and non-print) available to the schools, ^he efforts of 
\« non-commercial curriculum development teams have been swamped by 
this increase and their many expensive product ^^^^i^P^^^^^^^^^^. , 
learner verification technigues have been ignored by the commercial 
Producers who found the procedures unfeasible and too costly. The 
ox??a supply of funds fo? instructional materials induced Publishers 
4 tSrIS'SoL products on the market (where sales ^^^J* ^^^^ 

no thought of instructional effectiveness. California and Florida 
have led th- way in requiring that new instructional materials be 
l:?::ri:r verifiL"; pe?haps this might lead ^^^^ "^^^fj^^^^^.^^^^rd 
development that utilizes common-sense empiricism, small-scale tieii 
t^His and accuinulated experience. Such evaluations are greatly 
neeled'whon 99 percent of all instructional materials have not been 
verifi^^ by a 5>inqle lf^arn«?r. (WH) 
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The "so-BO yoars" a« over. Iheit positive promise has not b*.en reall«d. 
their negative side effects persist, and rheir .ost useful residunl contribu- 
tion goes largely ignored by those who have the .ost ro r.aln from it. 

These "go-go. ^r.ears" " the. late 19f.0s through the old- 1 960s - "" "^.^^ [ 

birth of programed instruction and the uobilization ot a phalanx of nultlnil- 
lion-doUar curriculum development projects funded by Federal and foundation 
support. These projects, it was thought, would overpower the tradition-bound 
lnstructlor.al materials market with exemplary, innovative products and thereby 
change the product . development practices of the education industry. Zealous 
to a fault, the curriculum reformers of the "go-go years" talked about how 
their materials would be "school proof" and "teacher proof" even as they were 
spending millions of dollars to train teachers to use the materials properly. 
The tacit strategy of those years was to use curriculum materials as a lever 
tor changing the "what" and the "how" of teaching and learning in elementary 
and secondary schools across .he country. Tl,e strategy did not work as well 
as planned. 

Kow, at the end of an era. we a.sk why its positive promise is largely ur.- 
real^-ed. And why its negative side effects persist. And why its i..ost ^ful 
residual contribution is ignored. Let's coasiJer these questions cne at . tirr.e. 

Sirply nut, a large part of the positive pronxse of the "ZO'Z^ years^ was 
to i-rove tho in.struct ional cf f uctiveno.3 of the cclnrational oro.iu.t. u..ed bv 
tenrV....'s . learners. Dut for tho con-^ercial publisher, who supolio. ro.t of 
tiK pnviurts. tc follow tb. dcvel .r.^nt-.l procedure, of the :P.r..vati-.'0 pro- 
3..t. wo.:la invnlv. a lov.el of I inane in. that v:ouh! be .nbearnble. In p .rti.u- 

.-. . ■ . li..Ki-t:.st.-' :'.-ir ' .t.ri.:-. - n . sr^lo th.-t O.u- l -/ - 
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And so. almost 20 years and qufte a few miiUoas of ta:< dollars later, in- 
^.tructlonal naterials producers concinue to develop naterials in nuch the same 
way as they have always developed naterials. Axid so, the promise Is unrealized: 
InstruAional effectiveness is still aot the tnaior factor in the developrr.ent of 
the r.aterTals u^ied'to teach the country^s 50.000,000 school-n^ad children. 

Our second question — about the continuing negative side effects — brings 
up a disturbing ir-.ny. The very same Federal legislators and policymakers 
those who legislated for and administered the funding of curriculum projects 
designed to produce highly elfective instructional materials as models for the 
burgeoning education industry - were at the same time also passing laws (lob- 
bied for by educational producers and the educational establishment) which gave 
schools the money to buy indiscriminately any educational products, including 
those developed without instructional effectiveness as their major considera- 
tion. As Federal monies became available to schools for instructional mater- 
ials, the industry responded by developing even more instructional naterials. 
So, the effort to encourage the development of better materials was overwhelmed 
by a simultaneous effort to encourage the development of more, then stm riore , 
materials. It is this unfettered proliferation of rnore rntb^ than in- 
structional n:at:crials that is the concinuing ncsative side effect of the "go- 



p,o years." 



?...r«re t o-.'in- on to our third qu->.stion. hovev^r, it in Lr.porf.mt to urrder- 
..L r.-- fwllv .-.^-;ut tlv.- iP..-re:i.sir.K. rar.nlLude of the instructional p.itorlalT 
11. - bc-r.r. L.) hu::.-, .:ur:-.; tho.;H "r,o-.'/. years." and r.till c '-.rinv.es its ra-pr- 
.■rovta. This r.Mt si:-p!- to Jo, uhich r.ay L^:<vUin wV.y c?:..velur>erc. sellers, 
.,vl..ctor.., r.vA users c-. in^tr-M-t iona 1 :o.cerials hnvo toruied to avoid even trv- 



3 ^- 



have focui^ed on film media, some on textbooks, others on television or nulti- 
nedia systenis, etc. But, it has only been during the last decade that attempts 
have been made even to catalog systematically all instructional materials in 
all media. 

These decade-old efforts at cataloging are best realized in R.R. iiowker's 
El-Hi Textbooks in Print (for the print media, although listings are also given 
for nonprint materials which are directly correlated with textbooks) and in the 
catalogs of the National Infomiation Center for Educational Media (NICEM) at the 
University of Southern California (for the nonprint media). If one takes the 
1974 edition of El-Hi Textbooks in Print together with the ens of separate 
media catalogs of NICEM and NICEM* s monthly "npdates" publisi . i in 1974, it is 
possible "to make a reasonably good start at defining at leas^ the quantitative 
dimensions of the instructional materials now available for school use. . 

The numbers by themselves are instructive, but of equal interest is the 
overall multimedia profile that can be created by arranging the various m-,dia 
in quantitative order (as shown in FIGURE 1). This total of more than 300,000 

FIGUP-E 1: Cor.nercially Available Instructional Materials in 1974 
80,381 Uvsn filns 

54,632 35n^.ra filmstrips (sound and silent) 
/.4,762 overhead projection transparencies 
2 3,590 audio tapes and cassettes 
21,508 35rra slide sets 

19,532 Srnn filmloops and cartridges ^ , , x 

18,500 tcxthooV-s (including workbooks and pro^rar-d books) 
17,500 record -J (estir.atcd) 
11,790 vidoncnpes and videocassettes 

e',000 muUir.icdia kits (estiratcd) 

9,000 Rnrr.es and simulations (estimated) 

300,195 . 
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Un«3 way to appreciace fully what this figure means is to compat'e it to a quan- 
titative profile of the sar.e naterials available in th^ aarly 1950a prior to 
the "go-go years." 

If . we -go back- to the early. 1950s, the years daring which £l- .Hi ToiXSbQo'/.s 

i. n Prl^t began to appear as a separately botmd catalog*, we get the estinated 
profile that appears in FIGURE 2. This estimated increase to more than 300,000 



FIGURE 2 1 Comaerclally Available Instructional Materials in Earlv 19503 

(estimated ranges) 

9,000 - 12,000 16inm films 

' ' 4,000 - 6,000 fllmstrips 

/ 5,000 - 6,000 textbooks 

( 1,000 - 2,000 records 

19,000 - 26,000 Total range 



materials from 19,000 to 26,000 naterials gives a reasonably accurate picture of 
how the instructional materials market has changed during the last 20 years. 
Moreover, it clearly illustrates the trend of "more, then still more" materials 
that began during the 1950s, continued through the 1960s, and today shows no 
sign of abating. These l.ist 20 years have seen the number of textbooks avail- 
able to schools increased by 200 per cent^ the nunber of 16ran filns by 600 per 
cent, records by 700 per" cent, and filmstrips by 800 per cent. But, as starch- 
ling as these increaaes are anong siSch lon^-established instructional inadia, 
there also has been an enormous proliferation of totally new r.edia within the 



* One indicator of ho"./ ruch snallor the instructional nateri.ils narkt?t vas 
is the fact that prior to tho 1950s, E^l-Hi To: >:thnok.^ in Print appeared as 
a onC'?-c^-y^*•lr a]'p»r.'i:< to .'t '^»irv:lf^ i'^r.ue of the r v,,.i::in:* » \ 
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instructional tnarketplace. As FIGURE 2 indicates, the impact of audio tapes 
and ca:,.^tfttes. videotaped and -cassettes, overhead projection transparencies, 
jnulcinedia kits, 8ixi filaloops, and garoes and simulations had not yet hit the 
school market in the early 1950s. 

Today, in addition to the more than 300,000 separate instructional pro- 
ducts available to schools, there are Increasing numbers of larger, composite 
entities known as instructional "systems," which can in many different ways 
incorporate many media. However, each such system is itself a product. At 
times, these systems — in contrast with connaercially available instructional 
materials — may use locally generated ("home-made" or "found") materials, per- 
haps built around an instructional staple like a standard textbook, or locally 
developed curriculum guides that usually do not become knovm beyond the school 
or district. However, it may be vorth noting that recently a few commercial 
enterprises have sprung up designed to seek out and distribute such "home- 
grown" products and systenls on the commercial instructional materials market. 
At present, such systems number in the hundreds, rather than the thousands. 

• But one, as yet unnentioned, type of instructional material used by many 
schools chat does indeed number in the thousands is the so-called "freebie"— 
materials that are available to schools i^lthout charge, often on a loan basis, 
fror hi:-.dreds of business nnd industrial corporations. If we add these "free- 
bies" (one service lists rore thnn 20,000 in their r.^idos) to the nore than 
300,000 commercially purchasable r.ateridls now on the narket, vc can reasonably 
cbti' ito that tho avGrat;- state or local education n^oncy cov^rinj^ the nornal 

Ar.^de r.pan hns ncce.^js to so-t-where in the r I ".hbo-.h -.^ 1 Vn0,O0O aupnr.ite 
IP r-r.:r io:Ml !■ .■ r i-i 1 . .'n<' : <-^rp p.T>- r,v th-/ 
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r:.U that «.ra we able « calculate it with precision, th. perc.n- 

srowth ia th. number o£ product options available in the total instruc- 
tional materials n^rk.t froa the Mid-1950s to the .id-1970s .ight he a. high 
as 2.000 per cent. But let us be conservative and consider only those commer- 
cially available naterials that were in fact cataloged in the early 1950s 
(FIGURE 2) and those similarly cataloged today (FIGUBE X). "a would still 
come UP with a growth factor of 1.500 per cent! 

During the last tKO decades, then, in an ^nony that has outstripped the 
r«t of the world in providing its consumers .1th options w,ll beyond anything 
heretofore dreaded of. the instructional ^terials market has probably - and 
..V-o.t vithout notice - outstripped all other .arUets within that virulent 
c^nnony in the increase of product options. 

.o^parison. for Instance, such a key industry as autonobile ..anu-ac- 
turing has n^.t increased its product options by 1.500 per cent during the last 
25 years (even iT one includes all foreign r„dels, . m fact, one would be hard 
put to find any America. Industry of any sort which has e:.perienced comparable 
product option growth during the s=»e period or. indeed, during any comparable.^ 
period of years. 

IMS quantitative rheno.«no. d.serv.s a good deal of careful analysis. If 
onlv .>,.c.„:.., ..ch unproce.en.ed and rapid ,„antitative f,rovth provided an almost 
in..v:t.,b'.e cuarantec U-.t hUh quality, carefully developed. instructionaUy 
ca.ctlv.. ..teri.U ..culd be quantitatively over-^hel... d by rapidly produced. 
■,...s., Otcc.ive product options, tuu t.nd.d to particularly tru=^ for the 
,..:,r v.-.i.:!. ofr-r, s." l.;'-': b.-c.™s« tiv.'V wc-r^ n.-..vr. r.-thor t'.'in 



In such a cliP-ate, it is highly unlikely that cor^mrcial producers will 
spend the tine and nake the effort necessary to learn how to iniprove the in- 
structional effectiveness of their products. After ail, a coinnercial pro- 
ducer, unlike th.^ Federal- or -foundation^ fwded developer, is thinking about 

beating the competition to the market in order to recover and turn a profit on 
a $10,000 investnant in a filmstrip. or a $50,000 investment in a film, or a 
$200,000 investment in a major new textbook. 

At this point. we*re about ready to consider the final question: the 
inportant residual contribution fron the "go-go years." During the late 1950s 
and early 1960s, Jlihile no one apparently was looking, some of the Innovators 
began to use product development techniques that can be readily and relatively 
inexpensively applied to the instructional improvement of all n:aterials. These 
techniques constitute the useful residual contribution of the "go-go years." 
Not based on hard science, they anount to an accumulation of hard-won enpiri- 
cal procedures arrived at largely through the creative application of intelli- 
gent trial and error — reinforced by success. 

Devoid of the scientistic pretentiousness that characterized so t=any of 
rhe overstat e and overrold "instrucc icnal technolo^Ues" of the "fto-go" inno- 
vators, which nrn:M^.ed to ir.prcve a'l of education, this residual base of en- 
pirici.n yrc-^.i... only to help ir.r.-.ve th. i n.r.r.-.icn.l effectiveness of in- 
dividual instr".:ti("'r.?l nr.cri.tls. 



• 



approach to rrc.-iuct dcvalcpr„ent and ir.prover.nt cvolv.cl .slo-.;ly lut .il:^03t sir.- 
„U.P..ou.ly our of the :.-r;. of data-oriented curriculun rncorials J.v.lopers. 

. ..... . ^ t ■•• 

r. -.cnrUb:- t-cr tV ovol.-.fu:, of tl-o t.chniqw.s th.t ch.'..ract..r i.e tMs in.trnr. 
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Uonal empiricism came origlrvally fron research on learning, sooe froa the 
practice of teaching, others from industrial and military trainlnss. and a £e« 
fro» educational publishing. Although the value o£ thi3 empiricism has yet to 
be- broadly demonstrated with InstructtoMl . materials.M submit that its value 
would become quite apparent if the techniques it has generated «ere applied to 
significant quantities of the Instructional materials currently being used by 
teachers and learners • 

The present reality is, however, that only a very few of the «.re than 
300.000 materials that find their way Into the hands of the nation's 50.0.00,000 
school-aged learners have been anplrically developed and continually shaped 
for - and by - use with those learners. This reality exists, in part.-Jbe- 
cause of the slowness with which the educational materials industry introduces 
changes into the way it goes about developing materials. This reality algl, 
exLots because 30.000.000 learners attend schools which fail to seek out, to 
denand, and to select such empirically shaped materials. 

But first, let us .nake clear what is .ueant by a product which can be 
characterized as having been empirically developed or revised with instruc- 
tional effectiveness as a forenost consideration. The essential attributes 
„f any such product would not necessarily be apparent within the product it- 
:a£, hut rather would pertain to how the product has been developed or re- 
-...d. For ir.stanco, h..s the product developer systematically eath3red de- 
cao-i iin-.-r.r..inod d.ta? V.ar, the dc-veloper analysed those ,!,aa In order to 
.-.ccver not . n'v <.-hat .p..clttr.-,ll.v. learners have learned fro., the naterlals. 
......riftc.-.nv cl,oy i.vrc supposed to but did not i.'-i;n? 
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or nanipulatin,, relevant intarnal. or tuxtuai; v,.riables? Oftv^n. Such internal 
change, r.ay nean Uttlo more than correcting a ui:ileadin^ direction or replac- 
ing a coniusLa;> e:ca;.plc illustration wj^th a clearer on.. In other cases. 
r.ore. .radical i^urg'sry tnay bti in "rdu-r. _ 

As a furthet defining attribute, we should expect to find the producer * 
systematically gathering and analyzin, data about how the product is actually 
us*»d onc^ it gets into the hands of teachers and learners. The outcome of in- 
vestigations of these external, or contextual, variables that are affecting 
the product's use would nost often lead to changes in teacher's editions or 
manuals, directions to students, or chaiyes in (or the initiation of) inservice 
teacher training programs* 

Given this much of a definitxon. it is possible. I think, to use it to 
arrive at an estinate of what portion of the more than 300.000 currently avail- 
able co:"^erclal Instructional materials can be said to have been empirically 
developed with a demonstrable concern for instructional effectiveness held 



t 

clearly in mind. \ ,' 



' Such an esticate was first presented to a Consressional Subcommittee in 
1971:^. That 1971 estimate was based on a systematic sampling of products in 
the 11 madia cate,;ories listed in FIGURE 1. Howevrr. the total number of pro- 
di^cts cate.,ori3ed and henct available for sampling w.s at that tire sor.evbere 
over 200,000. It was estimated '.bat, of those 200,COO-plus rv.t.ri.U then 

* Th. coroleto text o£ ta..- "Statcnont of P. Kc-nneth Koroaki" and th_> 

n;uinK dinlorwith Con ;res..an P-radcma. aro .rinccd in H,; n rij^ I^^" f 



\.. - Ulable to schools, only ahout 2,000 cohld hav^ fulfilled one or nora condi- 
tionti of such a definition. As FIGURE 1 iilustracos, the nur.^ber oi • .iCori..:.i 
catei;ori2ed — and hence available for analysis — in ly/A ii> in oi 

100,000. This .lncr5«5« oc sope 100,000 products is hoc /l"^ .^^-i'^--"' . 

in large part, to the appearance of new products in Che instructional materials 

^rkec. A good portion of the increase is the result of more comprehensive 
cataloging efforts during the last four years. ^ Thus, oany thousands of the 
materials now accessible to schools according to catalogs are not, in fact, to- 
tally new to the narket, but simply more accessible to purchasers. But if, 
then, we a une chat the production figures for new materials by the edccatioa 
industry has been more or less consistent during the last decade, we may also 
assume with some confidence that the proportionate number of 100,000 additional ' 
products that might meet our definition would remain essentially the same. On 
that basis, we may update to 3,000 the 1971 estimate of 2,000 products whose 
development and/or revision had demonstrated one or more of the attributes of 
empirically developed and improved materials. Or an overall percentage of ap- 
proximately 1 per cent (i.e., 3,000 out of a total of 300,000): 

These numbers would seem to make devastatingly clear that the two-pronged 
strategy of the "go-'go years" — that is. the Federal and foundation fundin:^ 
o. o-.e-.r^l ^ry prc:'iiftr, on the one hmd, .nr.d of broad-scni.- purchasir- of r.-.id- 

-ri • o-.-'lv.~<-s the u-her ~ has had little irp.ict on th-J ivi:i^ri\c- 
ti.-;^.:l q-. 1 1 f^f t:'- - r.ij.^rity of products that find thuLr -.vny into ih" h.in.i . 
.3.- , rr. - io.!rn.. r/.. Tlio .-.h-tT qtmn.tiiy of ratcri.iLs in .-..^rvi.v.' usc- 

vhich in,:ruct!onal e£ feet ivt^r..'i5s "has never been n nrinary c-'.nccrn o.K.il;- 
ovcnvhcli-.^a the irp.;ct of th- relatively fcv; potent iallN?rore etTerM-ye 
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Civ.n th. ccality ol these nu..b.... the pro«nosi. Is not b.lght for rev«- 
.ta, this 20-yea. trend. Nevertheless, there is see evidence that the educa- 

v<„. ..n elfort if not to oove In that direction, at least 
tlon industry is naking an ettort. ii n i. , 

T_ 1071 tha Association o£ Anerlcan Publishers 
• to consider- such- a diMctlPn,,_ , In^ 11' 3,.,thl _ 

(Che industry's ™aJor trade association) conducted a precedent-settin, survey 

designed to discover the extent to which its .ember companies .ere engaging 

in product development of the type characterized by at lea.t one attribute of 

.he definition offered above. It was claimed by the respo-.ding publishers 

that nore than 50 per cent of the pro.rans completed durin. th. tvo ..-.rs prior 

1. A "fTf>1d tested In sotne way. 

to the survey had been tieia teavi-u 

fhTf such testing often "occurs after 
Although the report vent on to state that such 

tHe programs are published." this after-the-fact bcw to enplrlcis. should never 
Cheless be applauded by anyone concerned with improved Instructional product 
aevelop«e,.t. However, the reality of numbers st : 11 oa.es it hi.hly unlikely 
Chat the present .acerials-,uantity/inscructionai-Huality imbalance in the in- 
scructional materials .arUet will be redressed in the near future. For example. 
Che average number of new textbooks appearing in yearly editions of El-Hl Text- 
^ 1, mat durins the last five years has been about 1.000. But even if we 

nf a oroiected 5,000 to be produced during the next five 
assu-e that every one or a projecuea :>,uw 

years is .olng to be produced by only these ^.panics which rcspondel to the 

,urvey, this would .can that approximately half of t.«e new textbooks 
(i.e.-. 2.500) rlnht be co.ntod a..ons th. polential "rodressers" of the prese^^ 
q uan t i ty ' qv.n 1 1 ty irha lane o . 

AS important as it 'or these co.pani.s to do this (.nd ™e fervently 

•7 ^.^(\ nrDtUuT'; a-.v'iLii'jt u'bit: in live 
hon-. that they do), thp not i.-pv t - 2,..0 .ruuru r.. 

.Vrs. according to NlCEH's extrapolations, .ay approach a half-.ilUo„ .materials 



will hardly leU unless some other changes come about as well. 

One of these changes, of course, is for more producers to coitmit then- 
selves to builUins each of their materials upon a firm base of empirical 

•data— systematically , gathered f rom users — and to using these data to im- 

prove the product's instructional effectiveness throughout its developmental 
and full market life. Secondly, there is a need for state and local instruc- 
tional materials purchasers to establish purcha:5ing criteria that preclude 
the purchase of materials that have not been empirically developed cr revised 
(either texCually or contextually) in light of data gathered from users who 
have actually attempted to learn with the materials in question. Two states. 
California and Florida, have already begun to respond to this need. Both 
states have recently passed laws calling for publishers, when they offer ma- 
terials for adoption in these states, to supply evidence that they have engaged 
in "learner verification" of their products either during their development or 
their revision. 

But these two changes would tend to affect only new or newly revised pro- 
ducts. The reality of the instructional materials market today is that the 
majority of its 300,000-plu3 materials have never been empirically shaped by 
learner feedback during development; and. in addition, because most of these 
ratoriils ,:re fron the nonptint side of the market - which lacks even the 
rr .:.t:cn ci r-.-.ur=r pro.iuot rovfsion cycles - most of th-.se arc r-.terialG 
U acv.r l-.-.-n put tV-r.-.-r-h -^ny .>>i-t of ro^uU 'W '^chc^ul^d revision pro- 

T.lv.L.^ r. For ex.: 'n l.-.- , i.-.r- f ■ It :i .marketed to urhocls c.r3 tv^ldnn. if 
•••v.-r. r.-vised on th', of crpirical d.ita. either durinj; development or 

..aor th.lr ..Mponrna-- on th- i r.;t ruct Lon.l r.atori;.ls Pnr!:.>t. Of tha roro 
than 80.000 16miti films currently listed in the three-volume Index to 16mm films. 



published by NICEM. only a handful of them, fewer than 1 per cent, are iden- 
tified as having been revised since their original date of issue; yet many of 
these ftlns have been on the mrket for more than 15 ye.ira.* The reasons for 

this lack of developneritar or postpublication -r 16uaa films are r.any. 

some of which are grounded in tradition and others in the economics of sound 
film pcoduction. 

Nevertheless, in the industrial film field, instructional films do get 
developed and revised on the basis of empirical data related to instructional 
effectiveness. Sometimes, it's at the '•stor>'board stage." sometimes in the 
••roush cut stage." sometimes later. In other words, it can be done (although 
once a film is on the market, revision can, indeed, be expensive). 

However, it should also be noted that many 16mm films used by schools are 
not truly instructional. Such films may be designed to make a single, provo- 
cative statement, or to provide the viewer a look at a "slice of life." or to 
present an aesthetic experience, etc., all of which may set the stage for in- 
struction that is actually carried on by a teacher within a "contextual" in« 
structional design which may be supplied by the producer, but is at tines ei- 
ther purposely or unconsciously left to the teacher. Because such films have 
no internal design of inf.truction, the ey.ternal instructional dc-sign, if it is 
explicated ia in acco^-nany inz manual or other printed rratter, rav easily be re- 
vise-: to ir.pro-'.^ Lis instructional effectiveness b:..;ed on observations of learner 

* It should V. pointed out that the NICr- Tnd.x als. li.:. 16-:. iUr. that 

aro out of print. This i. du-^, in part, to th. lact tnat z.y. -^Cr- . data 
bank performs an archival function in addition Ic its lunction^as n ..ouru 

currontlv nvailaHlc '-atcrial.. However, rany ni the out-o.-prin. 
fil— • in-^----'c^ t>v *;TrF'' .iro far fr.^n "nut of use' by .••.cl-oo.s. ..^n/ 
tho'--- -uL-r'f-print ritl.s aro -till CLrci;]al'V? Lhrn.,, -h ill- ' ^. r->^. 
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reactions to a teacher who is followin:--, the instructional desisn outlined ia 
chu manual. However, if the film has no clearly apparent internal or "textual" 
instructional desii>T:i and is not accoupanied by a "contextual" design developed 

.... by -the 43roducar,.,it .iailg ..to^ fU5?s ..that properly lies outside 

the kind of films that may benefit from empirical instructional revision. It 
may be useful to think of such l^ium films and other similar media items as 
being astructional in their intent and to consider them as lying outside of 
this discussion. 

But, if we turn our attention from 16mm films to 35inm films trips, a med- 
ium which is almost always instructional in intent, the present situation, 
while in some ways no better than that for 16mm films, does hold some hope of 
future movement in the direction of correcting the present overwhelming mat- 
erials-quantity/ instructional-quality imbalance. 

At the present time, there are almost 55,000 silent and sound 35mm film- 
strips on the market, as indexed by NICEM. Unlike many 16cm films, however, 
a sound filastrip seldom stands alone. Usually, it is part of a series of 
filmstrips, and invariably the series is accompanied by printed materials for 
use by the teacher (usually a transcript of the audio, accompanied by discus- 
sion quiistions and sone sugy,G:iCions for using the series). On the average, 
th^'.re arc about four to five filmstrips in a series. This neans that the 
; 3,000 f ilr-strips currently availahlo Lu ?:chaols arc actually n-.ore like 11,000 
•.: ;i-retc« ir.-truct.i<nr:l •vnr. it i or, . I'acrh orT.iLy has L.'---a;ninable and analyzable 
jtrurt :t.n::3 ucquonce.^ . which — tmll'-'e rh*? f .is t-ru.min^, "nil of a piece" 
V nc-.-s of K-'T-n filns — c-m bo tov.r r.nl ] y nanipulaLod an;', revised with rela- 
•■ ..; •, -r.:\ r e 1 .' t i v'.' ' v lirrlo o--:!""-^ ^, if. ."i-.y sr."".' of J-vi'l upr-TT-.t of n.'.r- 
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Yet. an excen.lv. sanpUng .ade of thes« naterUU indicated that .one 
20 per «nt of those sanplod are note than 15 years old. and that only a nlnis- 
eule nunier have ever been revised in any way during the entire course of their 

r«her lengthy marVef life. But of all the -instructional ..edia .now available 

to schools, fltastrips - whether silent or sound are by far the r^st amen- 
able to revision Koch cextually and contextually. 

First o£ all. compared to motion pictures and to textbooks and other 
printed naterials. fllnstrips are relatively Inexpensive to produce and singu- 
larly less expensive to revise at any phase of development; and this includes 
postpublication revision. This is true because each element of a fllnstrip 
(either a separate slide or piece of artwork, or a discrete segment of audio 
tape) is relatively easily manipulated and edited, either In relation to other 
elenents or within itself. 

Further=.ore. production economics favor filmstrips over textbooks. Text- 
books require large print runs to reduce their per-unit cost and are dependent 
on large initial investments and large volume sales, leading to relatively small 
- often less than 10 per cent - per-unit profits. Conversely, printing costs 
per filmstrip do not drop as the sUe of a run increases. And a large volume 
sale of n rilr^j^trlp series hcr.ins at only 2.000 sets a year, but can return a 
prc'U r..;;ln, fron ?00 per cent to iOO per cent ard higher per unit sold. 

..•ithout the t..«bc,nU publisher', large product Inventory to hold him back. 
....its, bin i-.v:ti..l investment returned In relatively short order, the fUm- 
i In n -nc\ h-tter n.,sition to r.akc ch:<nr,eo ■'i'-.ed at lr:proving 
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slides or pieces of artwork or chose portions of the audio that require chang- 
ir-.>. Aad, if the chan.^e required aeens to sten frota a coatext-of-use , rather 
than ftw a textual, problem, the producer may need to nake changes only In the 

- pTlTlted material- used by. the ieachar. In, eUher case. the Investnent in time, 

..oney. and editorial effort is not great at all. Yet. the reality is that =ost 
■-.filostrips are not empirically developed and improved on the basis of data gath- 
ered systematically from learners. 

A similar set of economic and editorial conditions holds Just as true for 
the burgeoning field o£ instructional audio cassettes, overhead projection trans- 
parency sets, sound-slide series ~ even videotapes and -cassettes ~ as it does 
for filmstrips. Thus, on the surface at least, the flexibility of :hese never 
\...dia would seem to bode well for the broad acceptance of product improvement 
through empiricism, but the present reality is that this flexibility is not being 
effectively exploited t(5 that end. 

Another factor that nakes these newer, nore flexible .eHia pivotally im- 
portant for the future developt^ent of the instructional rnaterials market is the 
fact that they represent th3 growing edge of that market. The sound filnstrip. 
for instance, has denonstrat.d a sales growth durin, the last few years that no 
one could have predicted a decade a,o. In 1973, for exanple. .ales put th. filr.- 
.trip fl-.ld ^:h-vvJ of 16rP. .end r.otion p?cturcs for th.^fir.s^ -ti;;^^. ixi story. 

rr air.onr, the 70 ccrpar.ica that conoriue t'.e Educatrionnl 
,r. r.;.iril po'.C....^ ^;70.000,COO In s-^lcs for tho y.>,^r. The pro-.r.osis 
:.r Lh.- r..-.:t fc-.v yo.nr.. i.s ovon rore opLiir.ist ic Sales of audio tapes and caa- 
\ ,,,,es 10., a.;o passed the all bur stabilized sales of Instructional audio rocord 

m ov:-. .-.'IV .:!r.i-irlr L^Mc.:tnr "f the yxc:.^h of thlr. GC-ot-v.- c-: tV' : 
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o£ producers and dtscrlbutors actively involved In it. As can seen in 
KICURE X the RrowtU of producers and distributors In the audio tape and cas- 
sette market from 1971 to 1974 is 20 per cent. 

V. is lnterestins to ref lect on the extent tc WMch growth 1^ the audio -■ 
cassette market my be related to the growth "(better than 30 per cent) shown 
a»ons producers/distributors of multimedia kits which make heavy use of audio 
cassettes. This increase in producers/distributors of multimedia nat^laU 
„ay also be responsible in part for the considerable Increase ik the areas of 
study prints and charts, both co.«on components of multimedia packages, ^.'e 
can further speculate that the smaller Increase in the number of producers/dis- 
tributors of filmstrips (10 per cent) during the sa.e period nay be somehow 
linked to the f.ct that filmstrip producers had already entered the multimedia 
sector of the market prior to 1971. But this is merely speculation chat may, 
ic is hoped, prompt a more thorough analysis of the relatedness of these com- 
plementary multimedia sectors of the market. 

Two other, more competitive than complementary market sectors referred to 
m FIGURE 3 demonstrate Interesting comparative growth patterns. These are the 
motion picture film field (16™. 8mm. and kinescopes) and the videotape and 
-cassette field. Tl,e robust 30 per cent increase in producers/distributors of 
the vld.o nedlun compared ro the modest 2 per cent increase In the 16r., film 
sector, .nnd the absence of gr.wth shown In the 8r.^ film and klnescop. s.-ct.~s. 
.aV.e this cor,p.-.tlt>v. se...ent .f the market also worthy of a future analy.i.. 

,„ comparison to the flexible, dynamically ch.n,;l„s side of th. instruc- 
tional rat.rials ..rV.t r;presented by th.se newer .onprint m.odla. the tcKthook 



nutr^ber of companies involved in the production and distribution of textbooW 
has. indeed. reuKiined quite stable over the la^t decade and sales { ic>.ures 
have risen only slightly during the last three years - topping $500,000,000 

la 19J3.ior ..the f irst tiine in historv - but these record-breaking sales are 

more a reflection of inflationary ri^^es in costs than of significant growth 
within the induiitry. However, what is perhaps more noteworthy is the fact that 
while they have been buying a relatively stable number of textbooks for some 
years now. schools have been steadily increasing the amount of their modest in- 
structional materials budgets spent on nonprint media. Thus, in 1973. wb^e 
elementary and secondary schools spent $54^,000.000 on tCKtbooka. they spent 
$423,000,000 on other instructional nateriais. 

The really significant difference, however, between these tx.o expenditure 
figures lies in th» dif f icult-to-discem area of comparative per-unit profit 
margin for textbooks and nonbook nedia. As already noted, the textbook is a 
hiRh-volur.e. low per-unit profit connodity. Textbooks have sold increasingly 
uell during their almost 200-year history in this country. One ~ajor reason 
for the continuously increasing sales of textbooks over the last two centuries 
has been this country's continuously ^^rowin- school age population. But this 
factor — -..'hich has not only continued throughout this century, but which in- 
crc.'.^ed i;'.jrply after V.urU! Uar II and through the "fio-go years" — has now 
st.pp.-i. -nd, uc. uru cold, it uill pry..-.hly n^ve sUr,htly in rcvers. for ths 
th. cor.M-.r;-. I::..r:tly -.vh.t eifcct this ui U have on the textbook 
.......t ^::c ir.stru'-.:- il r. :lvri.il:. ...:i-W-t is din'icult to say, but it is 

h..r.-] to i!. ;;i:^.e that tnis cl ..n,;e in what ha.^ amounted to bnlU-in, see-inr,ly 
:,.Mr.v...toed .r.inual sales increases i.s not s^)ing to force a nni^ber of other 

. .,^1 1,., .K- •i:;.-'r >■! ' '•■•^'•••s of qmntirv, cm- .."! f f • , 

t!- - « .>iv. .iJ i:. {Tovfn.'nc o. inut ructi )n.ii ii.itorirJ';. 



FICUUK 3: rMici (Nonbook). I'riulucor.-i / Di '-.trlbu 

1 Q7T . 1974 (pt'rconta^-.Q- chanjja) 



audio tapes and cassettes 


283 


350 


chai:,t3. , „ ^ 


56 




computer-asssited instruction 


4 


10 C+i5U/»/ 


dioramas 


13 


17 (+3oA) 


Sxnra film loops and cartridges 


240 


250 (+ 44) 


films 


180 


190 


iSn^n films 


710 


726 (+ z/»; 


35nm • 70nim films 


61 


72 v+l**'«>) 


f ilTostrips 


498 


5^8 (+10/«} 


globes and maps 


80 




kinescopes 


38 




micraf llns 


15 




models 


56 


CO 'JV\ 


multimedia kits 


196 


Zoo 


overhead projection transparencies 


212 


210 (- !/>} 


records 


260 


itoU V U/o/ 


videotapes and - cassettes 


112 


145 (+30%) 


realia 


25 


30 (+20:-) 


si idcb 


290 


250 (-14:0 


study prints 


92 


120 (+30:1) 


teaching; nachinc programs 


69 


66 (- 4V 



sources of data: Audiovisual M-rkec PTace. editions c 
years 1971 - 1974 
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One of those chan-es nay well be the appe.irnnce of r.ore tightly targeted 
print tnedia surrounded by co-related nonprint media worked into ^ini-systeins 
aiqed at individualising and personalizing instruction. The production runs 
for such new print materials would be much smaller than traditional textbook 
runs (and core like nonbook media production runs) because such text materials — 
unlike traditional textbooks — would not be raarUeted as though thdy were ap- 
propriate for use by all students in a given grade. IChe enabling technology 
for such small-run printing is available right now (available, in fact, in 
three new competitive printing/binding systems). 

Given the possible ««!gfijence of such tightly targeted, small-run print 
materials within a smaller, increasingly competitive market forced to respond 
to continually rising educational expectations, there may well be a growing at- 
tention paid to improving the instructional effectiveness of such materials. 
The unprecedented 1973 publisher-conducted survey on the prepublication test- 
ing of textbooks nay well be the harbinger of movement in this direction. I*ow- 
cver, that particular survey seems to have been motivated more by a need to react 
to the recently passed learner verification legislation in California and Florida 
by scientifically proving the quality of those products about to enter the mar- 
ket by a sort of ex-post-facto empiricism, than by a commitment to emoiricallv 
l-^orovinc; ths quality of these products throu-h rcjiular pre- and postpublication 
v?rlf icaticn an-i rovision. 

r.'c-vortV.clo^f;, tc:-.tbook publishers followed this r.urvpy with a trn.]-:- a;^r,i-c- 
i/'t.ioa 5-, c." to: --^nt i^ijutc: «..'.ri> i.i ^.lucu 1.1. <-j t.t.,i.<-- 

^^.i':h propublicatioa and postpubiicaLion verification and rovir.ion. The r.tatc- 
:-..:.t, while LOO r..:.:h in rh- vein of a solf-servin- npolni^a, drUs .'ivu* evi.^er.ce 
of movement in an important area within an industry that is not known for rapid 
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change. More importantly, this statement by the Association of Anetican 
rublishers does put ir.er.bc^r. of t\w textbook industry oa record as to cheir 
intentiof.s in this regard, and the AAP^sponsored survey does provide base- 

■Xin^'data ag^ainat which to n^.sute the. future fuMiUreat , of this stated in^ 

tent. Furthermore, whether tho^e vho favor nonprint media like it or not, 
textbook publishing, despite its depressing economics, is - and is likely to 
remain for some years yet - the mainstay of the education industry. And the 
t^.xtbook is likely to remain for some tine to come the nuclear median around 
which newer instructional media will cluster. Finally, if textbook publishers 
arc serious about delivering upon their stated intent to verify empirically 
and to revise their products accordingly, and - what is even n-ore critical - 
if they understand what thi^ implies in terms of practical operations havirg 
to do with fine-grained data-gathering, hard-nosed analysis, and the conacicn- 
tious application of findings to instructional improvement of every (textual 
and contextual) aspect of a material, then, there may indeed be hope for re- 
dressing the current naterials-quantity/ instructional-quality imbalance. 

I say this because of the still pivotal role textbooks play in today's 
curriculum, and because, rather ironically, they are the least numerous of the 
r.3jor instructional media. The potential impact of. say. 2.500 new. empiri- 
cally •-■cTified and irproved textbooks or. n rarket filled with more than 300.000 
r.tcria:-.. r.ost of -.vich ara not of this sort, ray seem srall in;'eed. Hut if 
2,S00 r.u.h v^atcTials wore to .ppoar in the n.:<t five years, as wo oarlier :.r-ru- 
i..ud th.y *c)ulc!, rh y voulJ .•.rco,:-t f..r .-.r. r.any n.. one out of evory 12 toxt- 
biiolrs on the ruirliot. 

Tr. l rniio cowl! :..cc-:v .i rriti.M ..u. if it w.-ro tn rivr-^r.-' or. 
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alnust 30.000 n.r.print nat«lals of a new. l.p«ve.l and coarlnu^lly inprov- 
,blu -- sort, -niose 2.500 textbooks and 30,000 nonpcint ..uterial:. r.i.*t ;u.t 
prove to be onou^h product options fron which discrlmin.tlng school selectorn 
nisht find what they ..eed to Improve classroom instruction. «^ least to be.ln 
„ltn.' riae; the current quantity/quality tnbalanie ^ight iven dtsapps^. 

What are the chances? 

At the present time, the chances do not look good.' Evidently, legislators : 
in both California and Florida agreed with this prognosis when they decided to 
pass lavs requiring chat producers engage in the et^plrical-based practices of 
learner verif icatlon'and revision for all naterlals offered for adoption in 
those states. But now that these laws have been passed in California and Florida 
^(and numerous other states see™ to be ready to follow suit), a critical question 
teualns. This question Is not. as some think, whether the states will de«nd 
rigorous compliance .to the letter of such laws. Rather, it is the question of 
how producers and users of instructional materials will go about meeting their 
responsibilities under these laws. Ir.plicit in this question is also the ques- 
tion of •.•hether ver>- rany producers have a clear understandins of what is ex- 
pected of them under these laws, and of .how well they ate able to translate 
. th.se expectations into operational activities desl,.ned to change effecllvely 
V.V.. w:,v th.y have traditionally ■■.vcbp^d their products. At the p.-.sfnt ar.e 
i, .......rent .l,.t nnt v.ry n.^y ruMl.h.rs h r.-. cl.-.,r ur..i-.rst.,n Mr.; 

_ ... -i -n- -tiv. cv«-n to'.-.-'r art- U.Mr. - rop:; t--^ 



lu .a. cxtvnt Lias is to be cxpect.d, ^iv.n the fact that law. niandatlnK 
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the empirical loam.r verification of nat.rlals nr. not yet b.cked up with 
.pc-olflc resuXatlon. a. to uhat producers nust do in order to comply ulch the 
laws, this has b»en compounded somewhat because the Caiifomln State Depart- 
ment u^-Educa^ion^s. initial, lnterpietatlo,a.of,, that _,sta^ 

law has tended toward reducing the intent of the law to a demand that producers 
prove that their materials have worked well with learners. And then reducing 
that demand to a point where ataost any sort of "proof is viewed as acceptable. 

In some cases, this has sent producers to t^elr files of teacher and admin- 
istrator product testimonials or to the task of soliciting such self-serving 
testimonials. In other instances, it has prompted those producers who can af 
ford' it to engage in large scale, after-the-fact field-testing of i.at«lals in- 
volving thousands of students and hundreds of schools in scores of school dis- 
tricts. The resulting statistical overkill produced by such studies has then 
been packaged in inprassive (in one case full-color) "Icar.er verification doc- 
unents." which have been offered as evidence that learners have indeed learned 
from the materials so tested. However, to date, the majority of such documents 
has been found to contain more files of testinonlals than tables of r.aan gain 



scores. 



But the publishers .,ho h-we hurriedly prepared these documents cannot be 
blared for ni-,.in, the pnint. bnvln., been r,iven th. impression that uhat they 
u- co-?ly with CUfornL. law is to er^na thn thetX rnerials ir- 
r.uv.er than to is-rove co.nn.u^ h- «^ Sl^ UBi^- NoverthcXesa. 
.v,.n tho,:ch th. lnn:.u.,-,e of the CaUfornia law i:. r.u.te cryptic, it docs state 
t-..-.t "I.t'.raor Vc.r.ific-,tic.-n r.ar.r, the continco.:s and t:-ornu-;h cv.au. Llon of in- 

'l -; for ir e ■ : f-'t i v " • ♦ 
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-to improve the quality and reliability of (nuch) natcriaU" (.M',33 . S^P'-^-a 
9234). However, to data, California has £ouad it six.pK'r to require ^a-Iliirli 
proof of a nu^terial's .tfectivenes. in a statistical report- of a oncci::'.. fx.lci- 

test .than to r^^^^^^^ 

continue to be — improved from revision to revision. 

But it can only be a matter of tine before policy makers in Califomia and 
_the publishers serving the schools of that state realise the extent to which 
-scientific studies" involving large "national or statewide sa«.ples," "experi- 
mental controls." .and "scientifically valid proce.iures" (and conducted by hish- 
priced "indepen^nt" evaluation agencies to lend credibility) can be an enonrous 
taisdirection of monies. Such monies would be far better spent on small 3:^.pU, 
carefully conducted, intensive empirical investigations designed to identify 

the specific segments, sequences, sentences, illustrations, or words that are 
hampering learners from learning what the material is intended to help then 

learn. Ihe after-the-fact, overly large-scale study is reminiscont of the 

sax..e sort of window dressing for sales purposes that cones with unnecessarily 

slick production foraats^ 

in an obvious allusion to the latter syndrome, one educational co^p.ny 
executive commented, at a r.eeting of educational producers durin:^ rhis Last 
y.ar, that these days the industry is standing at the "corn.r of i..r Color 
Bo-aleva.d ar.d Cosmetic Avenue." Tr.is nay v:oll be true, in.c I ^;ouU .;v:br-.; 
at on altern. todays -- it is .tandin^ at the •■cro.;.ro .d. ; • : -.t:.:: 

-.Vrc5..;ay .^vi i.-nirci.m Trail." "ihe cleutror.ic .i,.n:H,.tn K-.-.-linr. .0 .: 
r;pcea Ilxprcway deliver computor-^enerated rv.ssa j-.-nraaU... i u ; lo .v.a.^ - 
,•'•0 c.,. i».v.' th.- toll .-n .r.sy v\-^. - ''o-./nhLH .11 ^h- ^•■v. • 
.„ the other hand, is open to everyone willing to make the effort and is .irinu 
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nn.U btld-et buc it V. a slow, upUiU clL-nb. 
v; very one s DUUi>fi^i^ 

■ 1 materials selectors (dsspUe theit own 
expressions of intent) and instructional .atorxaU 

' ■ ■ ,-, „3n.lng "t„- put tU. best po..tblu ^t«inla., into, tn. J^apds o, 

Statements about wanting to p „^„irt- 
,._3..> ..U. .ct. be ..,ilUn. to .ollo« slo„ upbill cout. o. .P.i 
... especUll. U U «cns .V .-action in tbe no- o. ptoauct opt.on . 

Oi.n t. .t.l. tein.otcins beW« o. pto^uce. ^ ' ^ 

ui t-^ cnpculate whether either party 
last two decades, it does seeo reasonable to speculate 

. • ^^,.,t flow even If there were a good chance o£ 
„ould seriously consider reducing that flow, 

this resulting in better quality options. 

.his is the situation that .i.ht prevail, if Increasing n^b.rs of 
.at.s. L effect, reduce the n^er of product options that »ay be bou^. ^ 
.th state funds by local schools by si»ply precluding the adoption of »at 

have not been empirically learner verified and revised. . is outside 
L ... Of possibility that in order to avoid such " 
..oducers and educational practitioners „ould jointly lobby in the . odd 
..doption states" a.ainst any such restrictions on their freedo. to sele 
riu.. ^f.r all. it ..s .St such a publisher-educational establis.ent 

, , 20 years a^o succeeded in getting Congressional legislation 
lobby that almost 20 >cars a^o 

. 1 r.I aoll^r. into th. n..rUctplacc and thus opened the .o 
Pa.sed which put Federal dolU.r. 

. ' . rh^ f'-rst place, -mc fact may veil be that 
.0 years" of option ,roUferalion .a th. f.r.t pla 

" r^.rh . even if it- bas been greater 

...... decades o£ unprecedented pro..,c. option .rou.b 

„ , - .=y =i.„ly not be cno,.,h to s,tis£y producers 

than that of cny other i..v.u.,..> > 

r.ni purchasers of in.tructionnl r..terl.as. 
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.aa lu.al .a.cto.. of naterUU take che i.p.«ctvc o£ ..piricism serL^uay. 
.W tha. th«a nakc cva.y olfcrt to «cev.ai„ before ..l.cti.S a pr.- - 

auct «h.tlK.r or noc its ptoauc.- has cnpirlcally i.proveJ U. W th.t thoy 
«c willing to buy onlv those products that have been improved by .can. o£ the 

l.axn.r verification- a„a„revi.lo» process, Certa^.ly, if mterials =el.ctors 

wanted to. thay could take such a stand. And producers. «l,o are notoriously . 
responsive to selector demands (especially those .rtth purchasing power), would 
have no choice but to con^ly appropriately by e^irically i^roving every pro- 
duct they planned to keep on the market. 

However, even speculating about such an unlikely development raises the 
important question of just how realistic it is to expect producers to tnitiaie 
and sustain such a broad-scale coHnnit^nt. Clearly. It ^uld be totally un- 
realistic i£ we interpret such a narket de^nd to .«an that every product n<^- 
under development or already on the market would have to be subjected to a 
large sample. eKperi^entally controlled "scientific" validation study conducted 
at great expense by either an out-of-house educational evaluation agency or a 
separate division within an educaMo. company Itself. Cost aWe would oake 
meeting such a demand completely impossible for all but the largest and most 
..ofitable companies. «ut even for them, the logistical, statistical-sampling, 
and manpower r^nagement probler.s would be norrendous. 

on the other hand, if .ach a demnnd were b:v..d on the e-.cpect.tion 
t„,u . ..oh product catering - or beLng revioe^ for - th. n.rket would be r.^u- 
Urly ""J car«£«lly trlv-U on .mil ,roups of learner, who nro tc,tud. ob-.erv.d. 
anJ c-.a individually interviewed in order to find out ju.t where, why. and 
Uow „ :..,ieriat is failing to help iho. Uam. then, there is no question rhr.t 
.... ,.,,:„.,.- ...r.-..e..U.,.,e.-. .■..-vy :...lcn.l . . on ■.he ,M-.-.;n ' h. 
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...on^ically .nd otftclcnay verlfiod. rovls.d, nnd i=,proved. And. vh.t ts 
,.,.orc. there 1. ev.ry r.nsun to believe that any r.Wen sn.ll-scal. verification 
and tevUion of a naterl.l conducted in thl. fashion would be apt to throw 

3uWta*r.uch.-l£not-re,-.-U«hc..o.th.i,xt^rn^^ 

of that naterUl as any given large scale study, which is reple^. with a nation- 
wide sampling of .learners and a risorously controlled e^n^erlnental design. But 
at a snail fraction of the cost. 

in fact, a number of education industry "insiders" have privately voiced 
the opinion that the cost of such small scale Investigations of a material's 
instructional effectiveness could easily be absorbed by any company now In the 
.arket. Md it could be done without cutting into profits and without forcing 
companies to make room in product development and revision schedules for such 
activities, but not without properly sensitizing and training editors and dot. 
analysts to the nature of the Job they are being asUed ^ do. These insiders 
have also remarked that were the education industry to adopt such developmen- 
...1 procedures on a broad scale, it would greatly strengthen its case against 
. the Internal Revenue Service's recent ruling which has questioned the validity 
of what the industry has been writing off a, research and development expendl- 



tares. 



in addition to veritying materials empirically on very small groups of 
. i, a;:-,.- important that from time to time the menhir, of such edl- 
r.rl.l/r.ateri,',s-..alysis te.,n. personally n.d systomatically observe hew 
..chcol. .r. .cu:.,ny uUng the materi .ls they have d.v.loped or revised. Such 
....,rlfication observations are nocessar,- in order to gather data cn which to 
, ",,.,-.tc..t.>V r.vi.irns nf . particular material. But. hero 
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vorificadoa can probably l-e d..,. quit. effecUv.ly within a f«v, well-sel«ctod . 

classroons. The raason this sort of parsimony of numbers Is pcssible for both 

ccxloal and contextual learner verification „nJ revision activities is that. 

.unUke_.on..ttne .••scientific::, validation studias._ such act^ 

,ro gather fine-grained, detailed Uata that has high utility as . basis for pro- 
duct improvement. Thus, any si=e sar.ple of learners or learning settings - 
no matter how small - ^hat produces such data is quite acceptable. The ^mjor 
issue in product validation studies - predicting how well a product will per- 
foro with the total target population of learners, based on its perfor^nce with 
a representative sample of such learners - is simply a nonlssue for learner 
verification ind revision activities. 

The question of just how small a sample of learners may be used for such - 
investlsations can be argued, it one wishes to do so. but the answer to this 
question - as well as to many other questions related to the improvement of 
instructional naterials - can best be established empirically. 

Therefore, when a publisher cot-^ts to the continual learner verification 
and revision of a product, it is llV.ely thnt even one member of the tarset £oo- 
ulat urn SSL ^ S u.eful subject for a verit icatl.m ijilsode. This "so^nple" of 

is entirely acceptable ber..„s» It is reasonable to assume that any instruct 
tl...n,l Jifficultlo. with a ...i-.vn r-.torlnl that ar,- .xnerienc.d by on. ...mber cf 
the i.,r .-r :..r:..c.t .. uUKi -n .r. v,.rv UV.cly to b. ...-U.^.c^ by other re-b-.r. 
..f tha ; ;i.'at<,.n. r:.-:-., tiu- "l.T,. r,f parBi,:-ony" r,-.y P-rba,.. r^r,. eJc-,.ar.tly 
,,,„n.. I„ f,:. lc.r.....r v.ri,ic..t:.-.: r-vi— 'h.-n ..V. ..re el.c I. 

cducati.-r.,,. research. Ihls »h.,ul.. h. re.erberod by ul who nr.- tomptec to think 

, ,, . ,r'j- .-^ .1 ■to: rriT nf lar):e sirnl-> Ic.irncr vcrif:..--i- 
tion and revision activities neces.s..r i ly has a better p rogr an, ^ust because he 



— 28 — 



larse sanpR-.. .W ..y sinply b. wasting nonejr tl«e. and P^.l.* th. 



il o£ that v-Mst.! on to thn -school? 



But why. then, do nany companies in the education Industry .ngaga in large. 
■■nationarsa.pVe" aeia-t«ts and vaUdatlou .tudi^^^ 

a conpany's costs7 Vfl>y put so .uch .oney and faith In statistics^ Is It be- 
cause producer are cc-.itted to a policy of scientifically provin, the general- 
izAlllty of their findings statistically? Or has it «ore to do with the fact 
that a large national sample of school districts used in a field-test improves 
eventual sales statistics because It shows that "schools in your state" tried 

(and naybe even later purchased) these naterials? 

All companies haven't engaged in such a mixed use of statistics; nonethe- 

less, even nany of those companies now feel they will have to launch large sample 
■ field-testin, programs to m.et what they erroneously understand are the require- 

r^ents of the U-amer verification and revision process. 

one hopes that these companies have he... of the law of parsimony aod use 
It unsparingly. This Is not to s^a^ that all learner verMi-tlon eolsodas 
^ ^^cjr^^nc^ Hllhl^S3^M^- It doe. suj^. believer , tint a 

o^- learner e^sode wouM 

v^^. the cnniU^n tj-t .er.nins to the o ven.H.:nin, ^aioritv t^ mo- 
, M th^K-^nds of nafrl.I. currently belsa 22£L£.'M iS. SSi^. *»«> " 
Z^. condition ..ore th.n ..ny ntl-.r single factor that is responsibl. for the 
,:.....nt o,:.,...:'y/su.li-.y •■nl.....v if. the instruct iur.al materials rarket. 

It i. inf.-ror.tin,; n..te that the literature on instructional naterlals 



.. .r -( -.tw'lc-. tb.t have looUed at the effect of revl..lns 



Miterial cMi the bafWf; ui c .i>.i.v...i vci.ii^uwi 
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then, are studies by Flcning, 1063; Robeck, 1965; and ^Urkle^ 1967. The study 
by Robeck, discussed by Eva Bakc^r ii. AV Comr.un lent ion nevjew (U'lnter, 1973)» 
compared the perforrunce o£ an instructional material that had beea revised 
on tha basis -of data Gathered from a-single sixth, grader to. the performance of N 
the orisiaal (aoa-leai;ner verified) version of the material c-.nd. in addition, 
to the performance of yet a third version of the san,e material based on data 
from a second, single-learner verification episode. The three versions were 
tested on three maLched groups of learners. Baker summarized the results as 
follows: "The performance of the two revised editions was significantly bet- 
tev than that of the prototype (i.e., the original nonverified, nonr^vised 
version) although the performance on th« second revision was not found to be 
superior to the first revision." 

Given this provocative finding, let us indulge in some provocative fanta* 
sizing, ^^'hat if the Association of American Publishers and the Educational 
Media Producers Council were to join forces to sponsor a numbec of recogni:^ed. 
Independent Instructional materials researchers -- .such as Ernst Rothkopf, 
Susan- Mirkle, Eva Baker, and Richard Anderson, to name only the most obvious 
to conduct replications nf this study, using a variety of instructional mater- 
ials? And what if thay all cunfirned RobecVs findings to the effect thit a 
sir.:lo lo..rner v-MiMction and revisior "episode" nro.Mice.s si j-.ni f icnr.tly better 
Kv.inir..; ih.-.u ti;^' n. -v.^r if i.-d .".i n.nrovi^.-d oriv,inal ;-.:t i.at a s.^cond s'r.Ue 
l._..aT...'r v.Titi.-^ti.^n nd revisio:) vpiso^e produces no [ r.r. L f i cant incror-t in 
le rn- . v.-r t':: • ' - "t v.-r::i.-.i --^nd r- v d vor:;ic.n -n .^.tulied i-Tn- 
cr^? n.on. arr.-d with thi:i ov^'^'.-nco, what if the two oriMni::at ions convin.^od 
,.a. h of th.'Ir rc.pectiv.^ ror.bers (this would n^an rho-.tt 95 per fo-.it of nil text- 
book publishers and produuera of other instructional materials) to carry out 
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...-.i^Ual v.rifi»aoa .md r.vi.I.n of uU nac.rlaU currency b.in, 

.,, ...ch co-.,anV on at isast on. ic^^ (training odi=n.3 ..ad mat- 
erials analvst. to cacr. out t^e wo.U ni^ht taVe so.e tine, but it really ==.ld 
-b. ao.. in £a.it.Iy-....rc-ord«0 ■rh.n.. oac. .uch ..cpiricany bas.d , l=a™ 

uhat if evary school in the country agreed Co 
Avritied revision had been nada, what it <-very stn 

adopt and use these revisions immediately? 

U-Hat if all or this cane to pass7 It is as tempting to contemplate - as 
.ould be difficult to measure - the s-ss Increase in learning that -ight 
o.cur vith 50.000,OCO school-a.ed learners across the country d-aring the fol- 
lc.in. school year. Tt HSii Bi^ ^Hsiti- I-l.^ ^1 ^ 3-- 

^ infusion Hi c::oiric^ 
as iirmeasurable . 

And the only exceptions to the aingle learner verification episodes would 
,e cas.s in Which the single learner had absolutely no difficulty whatsoever in 
uad.rstandins and ...sterin. everything the materials in question was desi.^ed 
to t.ach. tn such estrone, positive cases, ..neralization to other laarr.ers 
..euld not he allowed and the verification and revlaion would be repeated with a 
learner. in ether words, .enarali.ation would be .ade only fron those 
. .,e le.m.r vrif ications in which t.,e Ic.rner e:.:perlenced difficulty in 

,'^,..„ W„.t ,!u. nator.aU w.ro Jesi^aed to h.lp students 



, ti.,, of •■■.d, .,id.s...le textual rc-:i.icn Tor 

. ,., tK-,e .-..t.v.al,.. the t«d.=«ry could r.ove toward .ore Mnil....-ty P.rsi- 
. ..onc,.:..t-of-u..e le .rn.-r v.-rltic-tlon and revision based on -aualU fine- 
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aa open school, a iov income school, n high iacoao school, suburban school. 



op 
etc. 



This suggestion Is nade with an awareness that it may be ridiculed by 
those in the education industry who "do not wish to change their traditional 
:r.ethod3 of product development and revision and who my. therefore, purposely 
misinterpret the spirit in which it is made. But in the long run. such ridi- 
cule cannot dull the ability of the point that is being made to puncture the 
complacency and inertia that are keeping the industry from fulfilling its pro- 
per role in this society. 

After all, the point of this fantasy is to underscore dramatically the 
reality that about 99 out of every 100 materials now being used by teachers and 
learners have never. been verified and revised with evea one. single learner. 
If this fantasy itself were to be translated into reality, publishers might le- 
gitimately ask how they can pick just one learner and know that he or she is 
the right one. To such a question, a reasonable response would be that since 
publishers cannct hope to find the one learner who is representative, they 
shouldn't try. They should sitr.ply find a reasonably articulate, not too overly 
bright member of the target population and pay very strict attention to what 
happens •..•:n'n he- or s-.e r.^s the nnt.^rials or answers questions abcut then. 

It •.•.■•.:ld .-'^.^ V-' r.>n-;.>nahio to r-^int out th.it .'.-.co lMs cr.^ l.>.:r"-r 
•icati-.-i i.-v:.: - r i'- -^r. I l.-id l-^cn corpleU-1 pub! i ..he'-s ro^.lJ.. in ti , 

» • ... • 1-1 » ill) K'lilTitJl* Vc^IaI ici- 

f^-;- I : ; Ca I t> li--*.* 

tion and r.vi.i.n .;rouP .,i,ht bo optirun for varlou. type, of r..U.ri-^.!. aaJ 



U- trrii r-; . 
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now and to date there has beer> v^re bonbast than action on the pa.t of the 
education industry. But no« that both bonbast and action by the industry 
have inc'rea^ed lu response to mandates fro. legislators In two ..ajor adoption 
"etates. it renins to be seen what educators «lll do in response to- such^nan-. . 
dates. They can help make the. Into important first steps along the path that 
the education industry seems willing to move if the market demands it. Or. 
they can help reinforce the status 3-0 by failing to view these mandates as 
.eans for correcting the present materials-ouantlty/lnatructional-quality iu- 



balance- 



Each company in every part of the industry will be watching closely every 
direct and indirect comnunicatlon froo educators on this natter. Perhaps it 
is too auch to hope that the substance of such co^unicatlon would be that both 
parties know that they've got to do bettor jobs of developing and using instruc- 
tional materials and they agree to get on with it. It the industry, in partlc- 
„1„. were willing to start operating from such a premise, the results could in- 
,.ed be rewarding for all concerned. But especially for the 50.000.000 ulti- 
nate consumers of its products. 

Cne of the things that the Industry could do to help Itself and these 
U..rners would he to .t.rt cpenly cor«unlcuttng about cffoctivo product improve- 

'^^ u r^iffiriilt to brlr^-x about ^vxM corr.Mni- 
- rM techai»iu«^.>. course v. j^-t*- 

,,,,o„ ...Uhln an Industry that U as hi.hly coapetttlve as the education ir.du.- 
f... v. V,. .-e, but for v.acs th- only re.l ■■ccrnu.lcatlon" of n^w kno-.-lodf..- 
.'. .t..in that industry has occurred via professional job r.ohlUty fro. con,p,ny to 
c.-.-..;,n-. Tn thin u.y. new techniques (such ..s th.y h .v= been) hav., been trans- 



critic!,-. M „..,. to:-hnu,u.r, d..,l,a....i to ..Uvnnuo the st.nte of t,«ruo,l«n.-.l 
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rr.atcrials Hevelopntant and usa has over existed. Uafortunatay . the .loumals 
of educational research do not pprform that function. .\nd in ny e::p-r i^.^c- 
they are not looked to by prof es.ioaals in the aducation industry as relevant 
to their very practical research and developnient ProblatnS' 

UTiere do the conmdtted professionals in the instructional materials busi- 
ness turn to increase their oxm professional growth? Because education lacks 
a fully developed "hard science" of learning to guide instructional materials 
specialists, they must rely primarily on empirlcistn - infortned by what little 
hard science research in learning has thus far produced. 

However, if the education industry does respond to this "isiperative of 
enpiricism" in an open tnanner. then some appropriate, useful means of communi- 
cation will emerge to reinforce that openness and to record the empirical pro- 
gress it will t^ake possible. Perhaps an important first step toward bringing 
about such conx^unication has already been taken by the present learrier verifi- 
cation legislation in two states which requires that a producer document fully 
the learner verification activities enga^jed in for a particular product. 

Despite the facr that the majority of the docur.antation nade available by 
corr.panios so far in response to the 1 :r«s of these states has proven to l.e of 
the toach.r-testl^.e^niaI variety. few of these docunents h .v. told n rood -'■-•a 
.bout pL-c-.h..ct d..vc.i..^r:ent that up. I J he useful lu tho profession, ■.:ith...u^ r^- 
vealin^. p.r.y pr -;-rictr.ry ir.iorr.ation. i^e hopes Lo see r -r of th.ir .-rt of > 

tin:, r: tir- .-v.s or.. S.^-.^ •■.'.Kit -ets c'-^cur-enfori rny sou;^'. T'.^-^ 
tific .'nd ray ind^^ed act.Mlly approach beim; scirnti f io , h'lt we cm sh .nU 
.>:pec:t • ach of It to b. the r.^ports of intelligent trial nnd ..rmr and ..hit Dr 
Susan Markle has called "successive approximations to an Ideal standard"— a 



standard that r,ay nnt ha specified with precision for y.ar.. but r.ay neverth.- 
Ic-s be pursued vith passion and confidence by caraful, dedicated. ...p ir ici.ts. 

To those vhu n^ay t^nd to doubt the. powar. of etnpiricism done on a snail _ 

scale and based on careful observation of tacts, attencion to detail, and ^n- 
tcUigent trial and error - it ir^y be w^U to remember that, at present, it's 
the best we've got. And that proceeding empirically in the absence of hard 
science in order to inprove what we have as best we can is better than indulg- 
ing in easy scientist, that inproves nothing - least of all, honest productive 
ccmunication. 

perhaps, in tine, a fully developed "hard science" of learning will emerge 
upon which to build instructional materials that are far superior to any we can 
uow iuagine. Perhaps not. However, no significant breakthrough of this sort 
seens likely to occur in the near future. Thus, as the hoped-for .ove.ent to- 
ward enpiricisn. outlined in these pav.es K.ains Tvonentur,. we nay do well to rerr.ind 
ourselves of another period durin. which enpiriclsn awaited the development of 
"h.rd science." This was a 90-year period in English history that culninated in 
dt-riisn of the "i-.proved" stean engine by Janes Vatt in 1776. 

Vnlcs-. you .-^rc ..r.on:; those :;till under the nisapprehension that J.^rr.es Watt 
r.t'M raM.or ^h .n riMZ^^^A ^^'^ ^^^•'^ (U-arned. vory likely, fron an 

•. ^ntarv nr . ■! I --ok you u.^d year'o ....vO, you nre uell au-arc of the fact 
t -t stf..-. iiK., hM N^.n invenr.>d ..n:! r.-.od about a century before in borh 
- .nd i-. r.r-any. Th.se were rather inefficient d.-vico^ corpircd to Watt' 

e-.-n:-, but th.y Ji'' v.-rk successfuHy enough to pur.p water out of r^ir.e shafts. 
• . thine, of r.rc-at i-noortnnco to tho nif.in;', in.lu.^t ry <,r the Li — . 

r,ur:n-. th. four Kom-rMions K-t.-.n t invention of that .n'-l... and '.'att': 



......lUbK. on,..«.... V.ut I::. - o. ....... .,nd ... ..ulU u-..n U.cn a«a in^or... 

tU... with vh« •■..-,.d sci..ul.i." K-.>owloa.. he had av.U.hU Uo. ... then 

a..i:-n.a a. oaia.. thar v,.. ...d ..„„,h-.to pow« .th. industrial ..volution, 



At ..asent. the ea..ati.n industry i= desperately in need o£ its own c.- 
pitically improved designs for Instructional ^terials. ...eth.r such empiri- 
cally improved de.i.ns will ever beco™ sufficiently in£on..ed by "hard science" 
„ power an "instructional revolution" regains to he seen, ".ut in the nean- 
tire. those advisins the education industry ™ust avoid acting as thou.h there 
is science where none yet exists. Tt,at is the e.press route to scientist - 
a word which can he succinctly defined as "the use of the trappings of science 
.0 trap nonsciontists into believing things their c6n.on sense would otherwise 
recognize as nonsense.** 
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